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A reading program evaluation was conducted by the 
school district of Eugene, Oregon, durirg the 1976-1577 school year. 
Three wajor questions were to be answered: (1) Are the district's 
elementary reading programs effective according to the district's | 
reading scope and sequence? (2) What sakes effective programs 
. (3) If programs are not effective, what can be done to 
ke them effective? The committee agreed to concentrate cn- 
comprehension and vocabulary. The study was conducted through 
questionnaires, interviews, the study of standardized test scores, 
and classroom cbservation. The data were prepared ag 32 individual 
reading reports, one for each school, each addressing reading 
instruction, factual and demographic data; and the strengthe and 
weaknesses of each school's programs. The refperts were sent to the 
schools, and responses were invited. Major deletions or 
pisinterpretations were corrected, and staff.disagreesentse with the 
report were included as one-page additicns to the reports without - 
comment. (A sample report is included.) (TJ). 
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Thirty-two Varieties of Reading 


The PURPOSE of this paper is to present the results of a reading Aya 


ation conducted in the elementary schools of Eugene, deegon, during the 1976- , 


- 77 school year. Emphasis will be given to the logistics of conducting the 


ar 


study, the political implications and the actual outcomes of the data. 
Rarely does a study emerge full bloom. This study was no exception. 
On October 6, 1975, the school board of Eugene, Oregon, requested that a com- 
prehensive plan for assessing reading instruction be Seteibped.x ‘The first 
phase of this work required principals to prepare descriptidns of their 
schools' reading’ plans. These descriptions were collected, summarized, and 
reported in March 1976 to the board. After discussion of the report in April 
of 1976, the superintendent's staff Eecimmandeg “chet reading instruction be 
further evaluated. With this Site sy assessment as. a starting point 


¢ 


decisions were made that the evaluation would focus on elementary schools 


only. The next step was the development’ of a preliminary evaluation plan which 


was presented to the board of directors in May of 1976. . This was followed in 


. 


June with a work session between the Research, Development and Evaluation 


(RD&E) and the board of dipectors to inform the board as to the eraluaeaem.. 


‘techniques, ad in ‘the general area of evaluating the reading program, 


and extent of the evaluation. -In August the board officially appointed a read- 
ing evaluation committee with a general charge "...that g comprehensive study — 
and evaluation. of the district's reading. program be undertaken."" In September 
of 1976 the seven-member committee met for the first time. The seven-member. 
committee was appointed by the superintendent through the authority of the 


board. 


‘ ‘ 


A year later, this method of appointment became a crucial issue. Since 


“the ‘committee was viewed as an arm of the school board, a legal implication 


arose regarding the state's open meeting laws and the rights of the press to 
be present at committee meetings. Legal opinion indicated that the committee , 


could be construed as an extension of the board because it directed that an 
: 


evaluation be done, Within this definition, the meetings of the committee 


could be viewed as public meetings open to press. As a result, one member of 


the news media elected to attend several of the committee meetings. This 


had interesting implications that will be further dealt with in this report, 


.The ‘committee was composed of an elementary principal; a member of the. 


University of Oregon faculty, the district's director of educational services, " 


an elementary reading specialist, an elementary classroom teacher, a member ” 
of the school board and a lay member from the domeniey: who was active with 
the local adult literacy league. The district's evaluation specialist was an 
a officie eum the committee. The role of the reading aielineton com- 
mittee was to determine what woul’ be evaluated, leaving how it would be done 
as the caaponsibittty of the RD&E staff. 7 - 
During the /spring of 1976, the five largest school districts and’ state 


departments of education in 11 western states were telephoned and asked if 


they knew of any school district which had carried out an evaluation of its 


’ reading instruction program. Other than Title I evaluations, this survey 


resulted in locating only one school district which had carried out an evalua- 
tion similar to the one planned for this district. A review of the literature 


published since 1960 was made to find articles or books reporting on actual: 


reading evaluations carried out by school districts. A search of the 1960-68 


~ 4 
Educational Index listed 82 referentes relevant to the evaluation of reading 


ay, a ‘ 


instruction. An ERIC search of published literature since 1968 listed 49 


\ 


; Ps . 5 _“ 4 
rpeernce to reading programs..’ All published references were read. No 
articles or beoks describing actual evaluations were located through these 
procedures. 


+ 


At the first meeting of the evaluation committee, the RD&E staff told 


_ the group “that, they -had been unable to locate any standards as to what 


“| specified a’ good reading evaluation. It was further seated that the evalua 


tion group was unable to obtain aieengne. few its source documents as to / 


a 


which behaviors constituted the teaching of Feading. A list OE APREP RDM TAY . 


100 topics that ‘could be ineluded in | the evaluation were presented to ‘thes 
committee.: By January I, 1977, the, committee had reviewed the ‘Enpics, ranked 


then as to pplority, m aide cost estimates o sack topic, ani! hid decided to 


“study 26 questions. On Getdher 19, “1976, the ‘reading ‘evaluation committee 


stated three major Go to be. answered by the evaluation. ~ These were, ; 


1) dene the district's denentaty reading PROgrans effective according to the’ 


district’ s seedtne scope and Sequence; 2) what makes effective programs . aw 


wetietlive and 3) if prograns are not éffective, what are those SHADES that 


y ‘ 


sould be done to’ ‘make them effective. i The study of the 26 DEE CLONS identi- 


fied by the committee would take place within this threefold context. Since 


“ ” 


agreement could not be reached as reve gensesites the teaching of reading, 


the committee agreed to evaluate the following Kinas of. ie: 3 activity: ‘com- 


prehension, pacubit ely oral peading. word analysis, sey: skills and silent 


\ 
reading. It was not possible to systematically avaluails all of: thesé activi- 


‘ 


ties and in May of, 1977 the committee decided to concentrate on the first two; 
| 7 ‘ Pe 


= “ ; . rs J ; * 4 
comprehension and/ vocabulary, as measured by standardized, norm-referenced 


: ry : a gost oe : : . . 
measures. The actual data collection plan was to involve existing test files 
in the district jas well as questionnaire and interview data from a wide 


variety of individuals. - Questionnaires were sent to all 402 people who taught 
3 | | 
| oy . v. 
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a regulprly scheduled reading program in grades one theoush six, to all ele- . 
mentary principals, all elementary iibvaedane; and all special edication 
teachers. These ‘groups comprise 944 of the district's élementary staff who 

‘ ; a ea 
are concerned with reading. All 26 reading specialists and 193 randonly- 
selected classroom teachers were interviewed. “These groups comprised 47% 
of the ‘contivies elementary staff ssiesnadsnleh reading. In addition, 66 
randomly-selected teachers and their 1,100 students’ were observed at four. 
different + ines: . | . - 

; 4 


" As can be seen at this point, the problem became.,one of too much data as. 


‘opposed to too little data. es ge 


ose 


« 


Types of fier Goi tented 
, ‘This onic briefly describes the major types of data ealiscted and ne 
general usefulness of each type. An attempt was made to isolate the costs ‘of 
reading’ instruction through analysis of existing figcal data regarding expen- 
ditures at district building tevel that pertained to reading instruction. 
Several analyses were made of data from the district budget office, “the 
principals’ own estimates of exnenaituive, and the regional directors’ offices. 
The major result was that the accounting systems currently aad iy the dis- 
trict are not adequate to allow for this fine a determination to be made 
sere ee reading or any other ‘academic gragrans, , It was found that this por 
tion of the study took a dvepvoporitenate amount of wesourees as compared to 
the actual usable data produced by the effort.. Each reading specialist at the 
elementary levels (N=26) was interviewed.. These Sneenuies were an attempt 
8 e 
to determine the characteristics of reading specialists, the work they per- 


formed, their percept ions of the strengths, weaknesses and overall effective- 


ness of reading instruction in their assed school, how children are 


‘ 
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selected to receive help from reading specialists, and the specialists' 
2 > 


recommendat ign for improvement of reading instruction in the district. A 


problem encountered was that it was not clear who were reading specialists. 


mes 


The phrase (reading specialist" was used to describe those people who work 


in schools and provide special help in reading. When this problem was re- 


olved, it was found that 26 people serve the elementary schools as reading 


. @......... These data were quite useful in developing the final reports 


on each of the schools' programs. 

Another group interviewed were the media specialists. They were sur+ 
veyed regarding the interfacing of their role with the reading instruction of 
each school. Data were obtgé#ffed from all schools with a-media specialist 
assigned to them. The data were somewhat useful in the final write-ups about 
each school. 

All elementary teachers directly involved in the reading program received 

. | 
a 14-page questionnaire which focused on teacher evaluation of reading nater- 
ials as well as identifying eeeeiiais used, availability of resources, etc. 
The results of this questionnaire were very useful. 


A random sample of six elementary teachers involved with reading from 


each of the elementary schools in the district were selected for.individual 


> ’ 
.interviewing. The purpose of these interviews. was to obtain highly dpotteid 


| 


information regarding the reading instruction program. The interviews (ap- 
proximately 20-60 minutes each) focused on reading instruction for different 
groups of children and their recommendations for improving reading instruction 
in) their school and ro the district. The data from these intervjews were 

5 eee useful in developing the fiat separ. : “ 


( 


,. Classroom observation data were obtained on randomly-selected classrooms 


to determine how much time of the reading time block was actually-spent 


teaching and studying reading. A 25% random sample of all second, fourth; 
and sixth grade classrooms were selected for observation. Selected from 34 
schools were 72 classroom iechske and their 1,100 students. Regularly 
scheduled reading periods for teachers and classrooms selected were observed 
by trained observers four times during the spring of 1977. This portion of 
the study! was handled under a contract-with an outside agency. A coding 
system was developed between the contractor and the RD&E unit regarding the 
types of vehavior that would be observed. The time frame was such that these 
_ data were not available to meet the timeline of the primary report. Subse- 
quent study of these, variables has been of interest and further consideration 
must be given these data before their implications for instructions are 

fully utilized. . 

Standardized test data were used as an objective measure of reading, 
cmuatonstion and vocabulary. Initial intent had been not to use étandandized 
test data except as an auxiliary checkpoint. However, these test data were 

—_ 


commonly available across all of the groups of students in our elementary pro- 
e 


gram. These data became’ extremely useful in the school-by-school analysis. 


Demographic Data 


Limited demographic data from the 1970 census were available for each 
school attendance area. While these data were acknowledged to be seven years 
old, they did weevide a common basis across all schools as indicants ‘of mean oa 
income and educational level of the total population in each schay's atten- 
dance area. These tata were quite useful in helping to “interpret the total 
reading report for each building. 


{ts 


Development of the Report 


As the reader can now tell, mountains of data were on hand. The question 
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was what to do with all of it. .The next step was to systematically arrange 


the" data into ways that answered the committee's 26 questions. Going back 


to the title of this paper, "Thirty-two Varieties of Reading," one comes to 
the focal point of how to report the information. It would fave beeh an 
SASSAE TASK if one report could have been written for the district's reading 
program, but there was no district reading program. This was known in ad- \ 
vance and in part decesadnad the evaluation design. While there is a set 


of district reading objectives of K-12, each school was free to implement the 


objectives in different ways. .As a result, it was necessary to prepare 


- individual school reports; hence the development of a multitude of reports, 


one for each of the: 32 varieties of reading programs. 

Step one was to prepare individaal reading reports. These weve Linitad 
in nature, baling not over three pages in the final report. The report was 
to address a brief ‘feeeviveionat the dchiogtes isan instruction, considera- 
tion of factual and demographic data, and consideration of the suchiEets and - 
weaknesses of each school's program, An example of one school's report is in 
the appendix of this paper. The variety of data sources that were desnitbed 
in the preceding section were used in writing the description for each school. 

Following General Accounting Office procedures, the evaluation design 
had a feedback loop at this point. After each school's report was prepared, 
RD&E staff sent copies,of the report,to each staff member and scheduled an 
appointment to discuss the report with the staff. The ground rules for the 
feedback loop were that any obvious errors in the data would be corrected 
immediately, any major deletions or misinterpretations of the data would be 
cosgected upon receipt of documentation from the staff, and, if the staff 
disagreed with any of the interpretations, they were invited to prepare a 
one-page statement of their position. The RD&E unit would not edit or comment 


on the teacher reaction to the report, but would include it as part of the 


\ 
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/ 2 % 
document. ‘The feedback sessions were conducted in all of the elementary 
schools and lasted between 15-240 minutes. Those in RD&£ who conducted 
these feedback sessions agreed they were quite valuable--clarifications were 
made and, in some cases, reports were modified on the basis of it;gin rare 
occasions total resistance was found to any of the data presented. 

Step two was the incorporation of the building reports plus adgitioral 
data into a 280-page document entitled ''The First R: A Report on the Effec- 
tiveness of Reading Instruction in the Eugene Public Schools, Volume One, 
December 1977." This was followed by a 70-page Volume Two in March of 1978. 

| : A final step built into the evatuation was to employ two independent 
outside evaluators to spend two days going thaough our evaluation and state 
what were the major weaknesses within it prior to submitting the report to 
the school board. The outside evaluators did an ddnirable job. When the 


report went to the board of directors the questions raised by these evalua- 


tors were presented to the board. This was a reassuring and objective check. 


Conclusions 

Based upon the analyis of these extensive data, s@veral major conclu- 

sions were drawn. The summary of this report contained the conclusions given 
% rye : 
below. 

“Analysis of scores on nationally used reading tests indicated there 
has been dramatic improvement of reading’ performance at most schools in the 
district over the past two years, and comparison of scores from 1976-77 with 
test scores recorded by the same students two years before showed that 18 of 
28 schools had improved. Only nine had declined and one remained the same. 

On the average, students in the second grade classes scored in the top 


one-third of the nation-wide student population that took these reading tests, 


and fourth and sixth grade students are just above the national average. 


io 
8 


. »? 
Results of standardized tests and teacher judgment of the situation in 
the schools agree that. 15-20% of the children enrolled in Eugene's schools 


are experiencing reading difficulties. 


Five schools have more than 25% of the sixth grade students scoring in 


* the bottom one-third of the publisher's norming group. 


As a result of the analysis of school descriptions, some strengths and 
weaknesses of the reading programs were common to several schools. Commonly 
abearved strengths iictuded 1) development of a staff agreement to teach 
similar materials; 2) willingness of staff to work on improvement of read- 
ing skills; 3) help from persons specially trained in reading instruction; 
4) consistency in continuity between levels of the reading program; and 
5) ‘yse of the district's list of reading skills to make sure students ‘learn 
the skills they are supposed to learfi. Common weaknesses included 1) lack | 
of staff commitment to acenatiy out an agreement about the use of similar 
materials; 2)-no organized school program for reading instruction because of 
ineffective administrative leadership; 3) lack of materials; 4) not enough’ 
time spent on reading; 5S) not enough help from those specially trained in 
reading instruction; 6) a high and disruptive rate of students leaving and 
entering the program throughout is year; and 7) not integrating library 
resources into the school's instruction." i 

Overall, the major findings of this study would hardly surprise anyone. 
associated with public schools. The important aspect is that details were 
available, for the first time, on the reading program in each building. As 
a result, some major program changes are being made in specific buildings. 


The results and the mandate to do something about them has caused the ele- 


mentary schools to take a close look at their reading instruction program. 


a 


For example, some schools' staffs are now considering that the program 


in grades two and three does have .a direct relationship to the reading pro- te 
gram in grades four and five. A limit ber of schools are having to come 
to grips with the fact that regardless ‘@f what they're doing, it evidently 

Asn't the correct program for the students enrolled. While most of the 
schools came out iooking ony good, there were a small number that were 
identified publicly as having major problems. The summary of the report 

ae would not make many people very comfortable, but one could question 


if the results of an evaluation should make everyone comfortable. 


Reactions 
At this point, let us ee question of public and staff reaction 

to reading evaluation. Approximately a year before the Kenube was publicized 
I wrote a prediction of what the public reaction would be to this evaluation. 
My comments (that I'd had pinned to my bulletin board for a year) stated, in. 
essence, that there might be a total of one dozen phone calls about reading 
programs from parents to any one school} there would be up to a maximum of 
three letters about our reading program in the local newspaper, and teachers 
or some staff groups might obfect to the report and/or attempt to discredit 
the report. My prediction was almost true. I don't know how many telephone _ 
calls schools received, but I haven't Head there were very many. Independent a 
observers of the letters to the editor column found zero letters pertaining 

to our reading program. Some teachers and principals were very perturbed by 

the findings@of the evaluation. The local education association did request 

that the state and national. education association examine the report with 

the result that a research syaciuliet from the National Education Association's 
“Washington office was brbiipht to Eugene to review the report. 


Earlier in this paper reference was made to the open meeting law of the 


state of Oregon. A representative of the Eugene Register-Guard, in attending. 


10 
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the meetings under the legal requirements of the state of Oregon, had access 
Wy oe ; 


. 


_to preliminary reports and discussion. During some of the discussions the 


reporter wrote articles about the proceedings, which were duly published. 


The headline of the first article was "Schools Fail at Reading" and had the: 


Wy FE puta 


net effect of causing the committee to stop talking. | When the committee saw 


‘what they thought were private deliberations being report@d in the newspaper, “ | 


. ; XQ ; \ & c ; 
they became extremely cautious and said very little. The second part was, 
that, prior to.opur outside evaluators re\ ae report and prior to the 


report going to our board of directors, t press released the report through 
the pages of a local newspaper, The net rehult - that, although the public 
became fiPoracd: 1s wae in part, misinformed, Shortly after the initial news 
coverage, the. press printed & retraction rega ing a limited amount of the 
data. ; a . 
In summary, this paper has presented proced es uséd for an pviluation 


; oe Pe 

of. reading instruction, types of data collected, the analysis, and mentioned 
e.g . ee. ae 

problems resulting from the open meeting law in Oregon.  ~_ * 


. . - 
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APPENDIX 


. 


I. Brief Description of the School's heading Instruction 
Based on data collected in spring or 1977; « apparently did hot | Ge 
have a current plan specifying how reading would be taught. A plan 
developed during the fall of 1974 existed, but it had not been recently ; 
updated. The two sets of materials most commonly ysen in thie ‘building aa } 
were Ginn 360 and a Houghton Mifflin series. The record- -keeping system 
was in operation and teachers indicated that they received from and paceai 7 
_ on to appropriate levels student eacaie. "There was no’ reatting specialist 
other than the specialist assigned to the Title I program. There had 
been no major change in materials used within the school during the last 
- year. A Title I program existed within the building for 1976-77 and em 
teachers indicated that students were working up to teacher expectations. 
Mixed reactions wcie expressel as to the accuracy of reading test scores 


as to measuring how students were able to read. 


II. ‘Discussion of Table 
-g,an average sized school, appeared to parallel the district 
, average values for most of the variables presented in the table. An area 
where it did not parallel was in ‘terms of reading achievement scores. 
The second grade: scores were below the district mean, while the fourth and° 
sixth grade scores were above the district mean. This was an inverse 


pattern to what was typically encountered at those grade levels. Comparing 


A 


re 


eS :, a TABLE 11-133 


‘ > School: . 
id id , 
a : School 
%, . x Ff Data District, 
‘ Characteristic Grades 1-6 Average ~- 
SS Se 
. ' 2 . ‘ 58 = : r 
_ Student enrollment Sata fOr (9 76al 7 oss saseicncweexeweds eebes 296 271 
Number of certified SLAF f VOI6277 sari saniwa rege ey ees tonegues 17.67 ° ‘17.49 
Number of books) reported, in 1976-77 fnventory........eeeeeeee 3,964 - 4,312 
Number of media specialists Working at the school eo 
expressed as XS of a. full-time position....... eee re ore ss Py Se qe 
. . cy - ° . oe 
Number of reading specialists working at the school 
"expressed as % of a FOL SEUNG BOS TION cies diwen a ed oe eee ar » os} 
2nd grade 1976-77 reading scores (expressed in percentiles).. - 66 \ \. 69 
4th grade 1975-77 reading scores (expressed in percentiles).. 67 57 


6th grade 1976-77 reading scores (expressed in percentiles).. 65 60 


6th grade 1976-77 reading scores for students who stayed 
in the same school since the 4th grade (expressed in 
POrCeNnt | 168s cccadsacusens Feet GakGtoi vitae ae timkeacceenues 75 63 


% of school's 1974-75 4th grade students scoring in the 
bottom one-third of the student population taking the © 
VEST OMETONA Wak dcsntens chow: exe peniGnendaweness Rioneaeeits 20.45 29.99 


% of school's 1975-77 6th grade students scoring in the 
bottom one-third of the student population taking the 


REST NUELONEY vende a nschagnas sanwaerinneeesisea seca vEsiee: ne * 15.27 
Score on Organizational Continuity Index-for 1976-77......... . 49 55.06 
Student withdrawal rate for 1976-77.......scsceseeseceseseces 17.36 17.50 
% of people 25 years or. older with 4 or more years of . 

college in school's attendance area in 1970...........0005- 11:21 24.48 
Mean income of school's attendance area im1970...........00- $11,213.00 00 
Average number of minutes per day in teacher's scheduled ~- ; 

reading period 1976-77..........c cee eeeee oa eaeedinee areas 58 30.08 
Average number of- minutes per day of reading instruction . 

outside teacher's scheduled period "976-77....... AS colic 33 51.69 
Average number of years each teacher has taught as of ) “s 

January 1, 1977. ewe eee were eecececce @oeaveeee Pease eeececeteeces i 14.29 | 12.25 
Average number of years each teacher has taught the 
. grade they are now teaching as of January 1, 1977.......... 8.46 6.98 

-Classload of average classroom teacher T97GH TT vvctie coco cana 22.38 23.93 | 
+ ‘ j 
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the last two years of test data showed that at all three grade’ levels 


tested, the group average scores for 


4 


had shown a marked increase 
of from 13 to 18 percentage points for 1977. , 
Several teachers commented that there were too many students in the © 
claSsrooms at .’ However, for 1976-77 the data shows Pawar stu- 
dents per teacher’ than the district average. in addition, a Title I 
program existed in the building, which ies dats staff above and eyene , 
that required for imptementation of the normal program. 
Ree a 5 \ 
| III. Strengths and ueanieeses ‘ ; : 
| A lack of unanimity or even simple inte existed in the data con- 
cerning strengths and weaknesses of the reading program. Within this - 
limit, two strengths reported were continuity of program and the experi- 
ence and quality of the staff. | 
It appeared that the teachers at thought they had a continuous 
program from grades one through six. Within the structure of the school 
there appeared to be a more cohesive program at the primary level than at 
oy | ‘upper elementary. Overall, there was conflicting évidence regarding 
conmuniteatton between teachers within the building, as exemplified by 
i séme teachers reporting that they did not know what other teachers in the 
| building were actually doing in the teaching of reading. 
IV. Comments by School Staff — 
"The ~. Staff would question sone of the assumptions of this draft 


description. Test scores reflect that students are generally above 


w 


the district average in reading, yet the draft description implies that 
there is a lack of adequate reading instruction within eur building. 

The majority of the staff believes that the strength of our reading - 
program is continuity and open lines of communication. We held a summer 
workshop in August, 1974, &nd developed a_comprehensiye peadiag’ program 
under the direction of ~ » reading gudeialist, The decision : 
was made to have co- -basals, Houghton Mifflin and Girmm 360, utilize the * 
District Scope and Sequence’ and the . Regton Reading Tracking Cards. 

an average sized school, appeared to parallel the district 
average values for most of the variables, except in the second grade, 
where class loads exceeded district averages" (274). This is the only 
grade that scored below the district mean and then, by only three points. 

Comparing the last two years of test data showed that at all three 


grade levels tested, the group average scores for had shown ‘a 


’ marked increase of from 13 to 18 percentage points for '76-'77. The 


staff attributes this seve to a school priority of total staff commit- 


"ment towards emphasis on reading instruction. Even though for '!76-'77 ~ » 
: the data showed school had fewer students per teacher than the 


' district average, the classload was heavier in primary levels and lighter 


at intermediate levels. “ - . 

The Title I staff involved in the building did not work-tn- the 
classraom's regular fending program and is not part of the building's staff 
allotment. A very well planned Title 1 Reading Program supplemented, but 


did not substitute for each teacher's normal reading program. 


, 


The "Staff hopes our input concerning tke naturesof our read- 
ing program wil] be considered carefully in analyzing this draft. We 


are confident that our program is stronger than is. implied. ", 
S 
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